Dynamically-Sized Data Structures on Data Flow Architectures

ABSTRACT

A data processing system for implementing operations that generate a dynamically-sized output is presented. The data processing system includes a reconfigurable processor that is configured to implement a first operation, a second operation, a recording unit, and a control unit. The first operation generates an output, wherein a size of the output is unknown during a configuration phase. The second operation receives the output of the first operation as an input. The recording unit generates control data that is indicative of the size of the output. The control unit that provides the control data to the second operation, wherein the second operation processes the input based on the control data.

RELATED APPLICATIONS AND DOCUMENTS

This application claims the benefit of U.S. Provisional PatentApplication No. 63/309,908, entitled, “Dynamically-Sized Data Structureson Data Flow Architectures” filed on 14 Feb. 2022. The provisionalapplication is hereby incorporated by reference for all purposes.

This application also is related to the following papers and commonlyowned applications:

-   -   Prabhakar et al., “Plasticine: A Reconfigurable Architecture for        Parallel Patterns,” ISCA '17, Jun. 24-28, 2017, Toronto, ON,        Canada;    -   Koeplinger et al., “Spatial: A Language And Compiler For        Application Accelerators,” Proceedings Of The 39th ACM SIGPLAN        Conference On Programming Language Design And Embodiment (PLDI),        Proceedings of the 43rd International Symposium on Computer        Architecture, 2018;    -   U.S. Nonprovisional patent application Ser. No. 16/239,252, now        U.S. Pat. No. 10,698,853 B1, filed Jan. 3, 2019, entitled        “VIRTUALIZATION OF A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/862,445, now        U.S. Pat. No. 11,188,497 B2, filed Apr. 29, 2020, entitled        “VIRTUALIZATION OF A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/197,826, now        U.S. Pat. No. 10,831,507 B2, filed Nov. 21, 2018, entitled        “CONFIGURATION LOAD OF A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/198,086, now        U.S. Pat. No. 11,188,497 B2, filed Nov. 21, 2018, entitled        “CONFIGURATION UNLOAD OF A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 17/093,543,        filed Nov. 9, 2020, entitled “EFFICIENT CONFIGURATION OF A        RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/260,548, now        U.S. Pat. No. 10,768,899 B2, filed Jan. 29, 2019, entitled        “MATRIX NORMAL/TRANSPOSE READ AND A RECONFIGURABLE DATA        PROCESSOR INCLUDING SAME;”    -   U.S. Nonprovisional patent application Ser. No. 16/536,192, now        U.S. Pat. No. 11,080,227 B2, filed Aug. 8, 2019, entitled        “COMPILER FLOW LOGIC FOR RECONFIGURABLE ARCHITECTURES;”    -   U.S. Nonprovisional patent application Ser. No. 17/326,128,        filed May 20, 2021, entitled “COMPILER FLOW LOGIC FOR        RECONFIGURABLE ARCHITECTURES;”    -   U.S. Nonprovisional patent application Ser. No. 16/407,675, now        U.S. Pat. No. 11,386,038 B2, filed May 9, 2019, entitled        “CONTROL FLOW BARRIER AND RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/504,627, now        U.S. Pat. No. 11,055,141 B2, filed Jul. 8, 2019, entitled        “QUIESCE RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 17/322,697,        filed May 17, 2021, entitled “QUIESCE RECONFIGURABLE DATA        PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/572,516,        filed Sep. 16, 2019, entitled “EFFICIENT EXECUTION OF OPERATION        UNIT GRAPHS ON RECONFIGURABLE ARCHITECTURES BASED ON USER        SPECIFICATION;”    -   U.S. Nonprovisional patent application Ser. No. 16/744,077,        filed Jan. 15, 2020, entitled “COMPUTATIONALLY EFFICIENT SOFTMAX        LOSS GRADIENT BACKPROPAGATION;”    -   U.S. Nonprovisional patent application Ser. No. 16/590,058, now        U.S. Pat. No. 11,327,713 B2, filed Oct. 1, 2019, entitled        “COMPUTATION UNITS FOR FUNCTIONS BASED ON LOOKUP TABLES;”    -   U.S. Nonprovisional patent application Ser. No. 16/695,138, now        U.S. Pat. No. 11,328,038 B2, filed Nov. 25, 2019, entitled        “COMPUTATIONAL UNITS FOR BATCH NORMALIZATION;”    -   U.S. Nonprovisional patent application Ser. No. 16/688,069,        filed Nov. 19, 2019, now U.S. Pat. No. 11,327,717 B2, entitled        “LOOK-UP TABLE WITH INPUT OFFSETTING;”    -   U.S. Nonprovisional patent application Ser. No. 16/718,094,        filed Dec. 17, 2019, now U.S. Pat. No. 11,150,872 B2, entitled        “COMPUTATIONAL UNITS FOR ELEMENT APPROXIMATION;”    -   U.S. Nonprovisional patent application Ser. No. 16/560,057, now        U.S. Pat. No. 11,327,923 B2, filed Sep. 4, 2019, entitled        “SIGMOID FUNCTION IN HARDWARE AND A RECONFIGURABLE DATA        PROCESSOR INCLUDING SAME;”    -   U.S. Nonprovisional patent application Ser. No. 16/572,527, now        U.S. Pat. No. 11,410,027 B2, filed Sep. 16, 2019, entitled        “Performance Estimation-Based Resource Allocation for        Reconfigurable Architectures;”    -   U.S. Nonprovisional patent application Ser. No. 15/930,381, now        U.S. Pat. No. 11,250,105 B2, filed May 12, 2020, entitled        “COMPUTATIONALLY EFFICIENT GENERAL MATRIX-MATRIX MULTIPLICATION        (GEMM);”    -   U.S. Nonprovisional patent application Ser. No. 17/337,080, now        U.S. Pat. No. 11,328,209 B1, filed Jun. 2, 2021, entitled        “MEMORY EFFICIENT DROPOUT;”    -   U.S. Nonprovisional patent application Ser. No. 17/337,126, now        U.S. Pat. No. 11,256,987 B1, filed Jun. 2, 2021, entitled        “MEMORY EFFICIENT DROPOUT, WITH REORDERING OF DROPOUT MASK        ELEMENTS;”    -   U.S. Nonprovisional patent application Ser. No. 16/890,841,        filed Jun. 2, 2020, entitled “ANTI-CONGESTION FLOW CONTROL FOR        RECONFIGURABLE PROCESSORS;”    -   U.S. Nonprovisional patent application Ser. No. 17/023,015, now        U.S. Pat. No. 11,237,971 B1, filed Sep. 16, 2020, entitled        “COMPILE TIME LOGIC FOR DETECTING STREAMING COMPATIBLE AND        BROADCAST COMPATIBLE DATA ACCESS PATTERNS;”    -   U.S. Nonprovisional patent application Ser. No. 17/031,679,        filed Sep. 24, 2020, entitled “SYSTEMS AND METHODS FOR MEMORY        LAYOUT DETERMINATION AND CONFLICT RESOLUTION;”    -   U.S. Nonprovisional patent application Ser. No. 17/175,289, now        U.S. Pat. No. 11,126,574 B1, filed Feb. 12, 2021, entitled        “INSTRUMENTATION PROFILING FOR RECONFIGURABLE PROCESSORS;”    -   U.S. Nonprovisional patent application Ser. No. 17/371,049,        filed Jul. 8, 2021, entitled “SYSTEMS AND METHODS FOR EDITING        TOPOLOGY OF A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 16/922,975,        filed Jul. 7, 2020, entitled “RUNTIME VIRTUALIZATION OF        RECONFIGURABLE DATA FLOW RESOURCES;”    -   U.S. Nonprovisional patent application Ser. No. 16/996,666,        filed Aug. 18, 2020, entitled “RUNTIME PATCHING OF CONFIGURATION        FILES;”    -   U.S. Nonprovisional patent application Ser. No. 17/214,768, now        U.S. Pat. No. 11,200,096 B1, filed Mar. 26, 2021, entitled        “RESOURCE ALLOCATION FOR RECONFIGURABLE PROCESSORS;”    -   U.S. Nonprovisional patent application Ser. No. 17/127,818, now        U.S. Pat. No. 11,182,264 B1, filed Dec. 18, 2020, entitled        “INTRA-NODE BUFFER-BASED STREAMING FOR RECONFIGURABLE        PROCESSOR-AS-A-SERVICE (RPAAS);”    -   U.S. Nonprovisional patent application Ser. No. 17/127,929, now        U.S. Pat. No. 11,182,221 B1, filed Dec. 18, 2020, entitled        “INTER-NODE BUFFER-BASED STREAMING FOR RECONFIGURABLE        PROCESSOR-AS-A-SERVICE (RPAAS);”    -   U.S. Nonprovisional patent application Ser. No. 17/185,264,        filed Feb. 25, 2021, entitled “TIME-MULTIPLEXED USE OF        RECONFIGURABLE HARDWARE;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,647, now        U.S. Pat. No. 11,204,889 B1, filed Mar. 29, 2021, entitled        “TENSOR PARTITIONING AND PARTITION ACCESS ORDER;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,650, now        U.S. Pat. No. 11,366,783 B1, filed Mar. 29, 2021, entitled        “MULTI-HEADED MULTI-BUFFER FOR BUFFERING DATA FOR PROCESSING;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,657, now        U.S. Pat. No. 11,263,170 B1, filed Mar. 29, 2021, entitled        “LOSSLESS TILING IN CONVOLUTION NETWORKS—PADDING BEFORE TILING,        LOCATION-BASED TILING, AND ZEROING-OUT;”    -   U.S. Nonprovisional patent application Ser. No. 17/384,515,        filed Jul. 23, 2021, entitled “LOSSLESS TILING IN CONVOLUTION        NETWORKS—MATERIALIZATION OF TENSORS;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,651, now        U.S. Pat. No. 11,195,080 B1, filed Mar. 29, 2021, entitled        “LOSSLESS TILING IN CONVOLUTION NETWORKS—TILING CONFIGURATION;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,652, now        U.S. Pat. No. 11,227,207 B1, filed Mar. 29, 2021, entitled        “LOSSLESS TILING IN CONVOLUTION NETWORKS—SECTION BOUNDARIES;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,654, now        U.S. Pat. No. 11,250,061 B1, filed Mar. 29, 2021, entitled        “LOSSLESS TILING IN CONVOLUTION NETWORKS—READ-MODIFY-WRITE IN        BACKWARD PASS;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,655, now        U.S. Pat. No. 11,232,360 B1, filed Mar. 29, 2021, entitled        “LOSSLESS TILING IN CONVOLUTION NETWORKS—WEIGHT GRADIENT        CALCULATION;”    -   U.S. Nonprovisional patent application Ser. No. 17/364,110,        filed Jun. 30, 2021, entitled “LOSSLESS TILING IN CONVOLUTION        NETWORKS—TILING CONFIGURATION FOR A SEQUENCE OF SECTIONS OF A        GRAPH;”    -   U.S. Nonprovisional patent application Ser. No. 17/364,129,        filed Jun. 30, 2021, entitled “LOSSLESS TILING IN CONVOLUTION        NETWORKS—TILING CONFIGURATION BETWEEN TWO SECTIONS;”    -   U.S. Nonprovisional patent application Ser. No. 17/364,141,        filed Jun. 30, 2021, entitled “LOSSLESS TILING IN CONVOLUTION        NETWORKS—PADDING AND RE-TILLING AT SECTION BOUNDARIES;”    -   U.S. Nonprovisional patent application Ser. No. 17/384,507,        filed Jul. 23, 2021, entitled “LOSSLESS TILING IN CONVOLUTION        NETWORKS— BACKWARD PASS;”    -   U.S. Provisional Patent Application No. 63/107,413, filed Oct.        29, 2020, entitled “SCANNABLE LATCH ARRAY FOR STRUCTURAL TEST        AND SILICON DEBUG VIA SCANDUMP;”    -   U.S. Provisional Patent Application No. 63/165,073, filed Mar.        23, 2021, entitled “FLOATING POINT MULTIPLY-ADD, ACCUMULATE UNIT        WITH CARRY-SAVE ACCUMULATOR IN BF16 AND FLP32 FORMAT;”    -   U.S. Provisional Patent Application No. 63/166,221, filed Mar.        25, 2021, entitled “LEADING ZERO AND LEADING ONE DETECTOR        PREDICTOR SUITABLE FOR CARRY-SAVE FORMAT;”    -   U.S. Provisional Patent Application No. 63/190,749, filed May        19, 2021, entitled “FLOATING POINT MULTIPLY-ADD, ACCUMULATE UNIT        WITH CARRY-SAVE ACCUMULATOR;”    -   U.S. Provisional Patent Application No. 63/174,460, filed Apr.        13, 2021, entitled “EXCEPTION PROCESSING IN CARRY-SAVE        ACCUMULATION UNIT FOR MACHINE LEARNING;”    -   U.S. Nonprovisional patent application Ser. No. 17/397,241, now        U.S. Pat. No. 11,429,349 B1, filed Aug. 9, 2021, entitled        “FLOATING POINT MULTIPLY-ADD, ACCUMULATE UNIT WITH CARRY-SAVE        ACCUMULATOR;”    -   U.S. Nonprovisional patent application Ser. No. 17/216,509, now        U.S. Pat. No. 11,191,182 B1, filed Mar. 29, 2021, entitled        “UNIVERSAL RAIL KIT;”    -   U.S. Nonprovisional patent application Ser. No. 17/379,921, now        U.S. Pat. No. 11,392,740 B2, filed Jul. 19, 2021, entitled        “DATAFLOW FUNCTION OFFLOAD TO RECONFIGURABLE PROCESSORS;”    -   U.S. Nonprovisional patent application Ser. No. 17/379,924, now        U.S. Pat. No. 11,237,880 B1, filed Jul. 19, 2021, entitled        “DATAFLOW ALL-REDUCE FOR RECONFIGURABLE PROCESSOR SYSTEMS;”    -   U.S. Nonprovisional patent application Ser. No. 17/378,342, now        U.S. Pat. No. 11,556,494 B1, filed Jul. 16, 2021, entitled        “DEFECT REPAIR FOR A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 17/378,391, now        U.S. Pat. No. 11,327,771 B1, filed Jul. 16, 2021, entitled        “DEFECT REPAIR CIRCUITS FOR A RECONFIGURABLE DATA PROCESSOR;”    -   U.S. Nonprovisional patent application Ser. No. 17/378,399, now        U.S. Pat. No. 11,409,540 B1, filed Jul. 16, 2021, entitled        “ROUTING CIRCUITS FOR DEFECT REPAIR FOR A RECONFIGURABLE DATA        PROCESSOR;”    -   U.S. Provisional Patent Application No. 63/220,266, filed Jul.        9, 2021, entitled “LOGIC BIST AND FUNCTIONAL TEST FOR A CGRA;”    -   U.S. Provisional Patent Application No. 63/195,664, filed Jun.        1, 2021, entitled “VARIATION-TOLERANT VARIABLE-LENGTH        CLOCK-STRETCHER MODULE WITH IN-SITU END-OF-CHAIN DETECTION        MECHANISM;”    -   U.S. Nonprovisional patent application Ser. No. 17/338,620, now        U.S. Pat. No. 11,323,124 B1, filed Jun. 3, 2021, entitled        “VARIABLE-LENGTH CLOCK STRETCHER WITH CORRECTION FOR GLITCHES        DUE TO FINITE DLL BANDWIDTH;”    -   U.S. Nonprovisional patent application Ser. No. 17/338,625, now        U.S. Pat. No. 11,239,846 B1, filed Jun. 3, 2021, entitled        “VARIABLE-LENGTH CLOCK STRETCHER WITH CORRECTION FOR GLITCHES        DUE TO PHASE DETECTOR OFFSET;”    -   U.S. Nonprovisional patent application Ser. No. 17/338,626, now        U.S. Pat. No. 11,290,113 B1, filed Jun. 3, 2021, entitled        “VARIABLE-LENGTH CLOCK STRETCHER WITH CORRECTION FOR DIGITAL DLL        GLITCHES;”    -   U.S. Nonprovisional patent application Ser. No. 17/338,629, now        U.S. Pat. No. 11,290,114 B1, filed Jun. 3, 2021, entitled        “VARIABLE-LENGTH CLOCK STRETCHER WITH PASSIVE MODE JITTER        REDUCTION;”    -   U.S. Nonprovisional patent application Ser. No. 17/405,913, now        U.S. Pat. No. 11,334,109 B1, filed Aug. 18, 2021, entitled        “VARIABLE-LENGTH CLOCK STRETCHER WITH COMBINER TIMING LOGIC;”    -   U.S. Provisional Patent Application No. 63/230,782, filed Aug.        8, 2021, entitled “LOW-LATENCY MASTER-SLAVE CLOCKED STORAGE        ELEMENT;”    -   U.S. Provisional Patent Application No. 63/236,218, filed Aug.        23, 2021, entitled “SWITCH FOR A RECONFIGURABLE DATAFLOW        PROCESSOR;”    -   U.S. Provisional Patent Application No. 63/236,214, filed Aug.        23, 2021, entitled “SPARSE MATRIX MULTIPLIER;”    -   U.S. Provisional Patent Application No. 63/389,767, filed Jul.        15, 2022, entitled “PEER-TO-PEER COMMUNICATION BETWEEN        RECONFIGURABLE DATAFLOW UNITS;”    -   U.S. Provisional Patent Application No. 63/405,240, filed Sep.        9, 2022, entitled “PEER-TO-PEER ROUTE THROUGH IN A        RECONFIGURABLE COMPUTING SYSTEM.”        All of the related application(s) and documents listed above are        hereby incorporated by reference herein for all purposes.

FIELD OF THE TECHNOLOGY DISCLOSED

The present technology relates to a data processing system, and moreparticularly, to a data processing system for implementing operations ona reconfigurable processor that generate a dynamically-sized output.

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves can also correspond to implementations of the claimedtechnology.

Reconfigurable processors, including FPGAs, can be configured toimplement a variety of functions more efficiently or faster than mightbe achieved using a general-purpose processor executing a computerprogram. So-called coarse-grained reconfigurable architectures (CGRAs)are being developed in which the configurable units in the array aremore complex than used in typical, more fine-grained FPGAs, and mayenable faster or more efficient execution of various classes offunctions.

Such reconfigurable processors, and especially CGRAs, often includespecialized hardware elements such as computing resources and devicememory that operate in conjunction with one or more software elementssuch as a CPU and attached host memory to implement low-latency andenergy-efficient accelerators for applications such as machine learningand artificial intelligence workloads.

Typically, applications are executed on the reconfigurable processors ina distributed fashion by programming the individual compute and memorycomponents to asynchronously receive, process, and send data and controlinformation. In the reconfigurable processors, computation can beexecuted as deep, nested data flow pipelines that exploit nestedparallelism and data locality very efficiently.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings.

FIG. 1 is a diagram of an illustrative data processing system includinga coarse-grained reconfigurable (CGR) processor, memory, and a hostprocessor.

FIG. 2 is a diagram of an illustrative computer, including an inputdevice, a processor, a storage device, and an output device.

FIG. 3 is a diagram of an illustrative reconfigurable processorincluding a top-level network (TLN) and two CGR arrays.

FIG. 4 is a diagram of an illustrative CGR array including CGR units andan array-level network (ALN).

FIG. 5 illustrates an example of a pattern memory unit (PMU) and apattern compute unit (PCU), which may be combined in a fused-controlmemory unit (FCMU).

FIG. 6 is a diagram of an illustrative compute environment in whichapplications are provided a unified interface to a pool ofreconfigurable data flow resources such that the pool of reconfigurabledata flow resources is available to the applications as a singlereconfigurable processor.

FIG. 7 is a diagram of an illustrative operation that generates adynamically-sized output.

FIG. 8 is a diagram of an illustrative reconfigurable processor that iscoupled to external memory and configured to implement an operation thatgenerates a dynamically-sized output.

FIG. 9A is a diagram of an illustrative buffer with a write operation toa first portion of the buffer and a read operation from a second portionof the buffer.

FIG. 9B is a diagram of the illustrative buffer of FIG. 9A with a writeoperation to the second portion of the buffer and a read operation fromthe first portion of the buffer.

FIG. 10 is a diagram of illustrative dependencies between read and writeoperations of the control unit.

FIG. 11 is a diagram of an illustrative implementation of anillustrative filter operation and associated control circuitry on areconfigurable processor.

FIG. 12 is a diagram of an illustrative implementation of anillustrative group-by operation and associated control circuitry on areconfigurable processor.

FIG. 13 is a diagram of an illustrative implementation of anillustrative unique operation and associated control circuitry on areconfigurable processor.

FIG. 14 is a flowchart showing illustrative operations for operating adata processing system with a reconfigurable processor.

DETAILED DESCRIPTION

The following discussion is presented to enable any person skilled inthe art to make and use the technology disclosed and is provided in thecontext of a particular application and its requirements. Variousmodifications to the disclosed implementations will be readily apparentto those skilled in the art, and the general principles defined hereinmay be applied to other implementations and applications withoutdeparting from the spirit and scope of the technology disclosed. Thus,the technology disclosed is not intended to be limited to theimplementations shown but is to be accorded the widest scope consistentwith the principles and features disclosed herein.

Traditional compilers translate human-readable computer source code intomachine code that can be executed on a Von Neumann computerarchitecture. In this architecture, a processor serially executesinstructions in one or more threads of software code. The architectureis static and the compiler does not determine how execution of theinstructions is pipelined, or which processor or memory takes care ofwhich thread. Thread execution is asynchronous, and safe exchange ofdata between parallel threads is not supported.

High-level programs for machine learning (ML) and artificialintelligence (AI) may benefit from massively parallel computations,where many parallel and interdependent threads (meta-pipelines) exchangedata. Such programs are ill-suited for execution on Von Neumanncomputers. They benefit from architectures that enable parallelprocessing, such as coarse-grained reconfigurable architectures (CGRAs)or graphic processing units (GPUs).

Reconfigurable processors, and especially CGRAs, often includespecialized hardware elements such as computing and memory units thatoperate in conjunction with one or more software elements such as a hostprocessor and attached host memory to implement low-latency andenergy-efficient accelerators for applications such as machine learningand artificial intelligence workloads.

Typically, applications are executed on the reconfigurable processors ina distributed fashion by programming the individual compute and memoryunits to asynchronously receive, process, and send data and controlinformation. In the reconfigurable processors, computation can beexecuted as deep, nested data flow pipelines that exploit nestedparallelism and data locality very efficiently.

These data flow pipelines contain several stages of computation, whereeach stage reads data from one or more input buffers with an irregularmemory access pattern, performs computations on the data while using oneor more internal buffers to store and retrieve intermediate results, andproduce outputs that are written to one or more output buffers. Thestructure of these pipelines depends on the control and data flow graphrepresenting the application. Pipelines can be arbitrarily nested andlooped within each other.

The data flow graphs may be distilled from a high-level program andtranslated to a configuration file for the reconfigurable processor. Ahigh-level program is source code written in programming languages likeSpatial, Python, Java, JavaScript, C++, and C, and may use computationlibraries for scientific computing, ML, AI, and the like. The high-levelprogram and referenced libraries can implement computing structures andalgorithms of machine learning models like AlexNet, VGG Net, GoogleNet,ResNet, ResNeXt, RCNN, YOLO, SqueezeNet, SegNet, GAN, BERT, ELMo, USE,Transformer, and Transformer-XL.

A Software development kit (SDK) usually transforms the input behavioraldescription of the high-level programs of the applications into anintermediate representation such as computation graphs (e.g., data flowgraphs, control graphs). The transformation from the high-level programsinto the computation graphs may include code optimization steps likefalse data dependency elimination, dead-code elimination, and constantfolding. The computation graphs encode the data and control dependenciesof the high-level programs.

The computation graphs comprise nodes and edges. The nodes can representcompute operations and memory allocations. The edges can represent dataflow and flow control. The computation graphs support branches, loops,function calls, and other variations of control dependencies. In someimplementations, after the computation graphs are generated, additionalanalyses or optimizations focused on loop transformations can beperformed, such as loop unrolling, loop pipelining, loop fission/fusion,and loop tiling.

A compiler transforms the computation graphs into a hardware-specificconfiguration, which is specified in an execution file generated by thecompiler. Thereby, the compiler translates the applications intoreconfigurable processor specifications.

While traditional compilers sequentially map operations to processorinstructions, typically without regard to pipeline utilization andduration (a task usually handled by the hardware), an array of CGR unitsrequires mapping operations to processor instructions in both space (forparallelism) and time (for synchronization of interdependent computationgraphs or data flow graphs). This requirement implies that a compilerfor a CGRA must decide which operation of a computation graph or dataflow graph is assigned to which of the CGR units, and how both data and,related to the support of data flow graphs, control information flowsamong CGR units, and to and from host processor(s) and attached memory,while aiming for maximum bandwidth and minimum latency.

Illustrative functions or operations in machine learning applicationsinclude non-linearities like Rectified Linear Unit (ReLU) and itsvariants (e.g., leaky ReLU), hyperbolic tangent, sigmoid, and softmax,element-wise addition, matrix multiplication (e.g., General MatrixMultiply (GeMM)), layer normalization (e.g., batch normalization), lossfunctions like cross-entropy, and tensor shape modifiers like transpose.Such functions or operations operate on input data that can includescalar data (e.g., control bits) and vector data (e.g., vectors,tensors, matrices).

Usually, such functions or operations produce output data of a knownformat and size such as a scalar or vector data of a predetermined size(i.e., including a predetermined number of elements, each having apredetermined size). However, some functions or operations may produceoutput data of an unknown size. For example, the number of elements ofan output vector or an output tensor may be undetermined at compile timeand depend on the input data and on the function or operation.

It is desirable therefore to provide a support for dynamically-sizeddata structures on data flow architectures. In particular, it isdesirable to provide a data processing system for implementingoperations that generate such dynamically-sized data structures. Forexample, the data processing system may include a set of techniques foreffectively handling an operation that is producing vector data (e.g., avector or a tensor) with an unknown number of elements at compile timeas well as effectively handling the operations downstream of thatoperation during the execution of the associated application onreconfigurable processors.

A data processing system for implementing operations that generate adynamically-sized output is described. The data processing system iswell-suited for applications like machine-learning (ML) and training ofneural networks and includes a reconfigurable processor. If desired, thereconfigurable processor includes arrays of coarse-grainedreconfigurable (CGR) units, which are sometimes also referred to as CGRarrays. Such a reconfigurable processor with arrays of CGR units issometimes also referred to as a CGR processor.

The architecture, configurability, and data flow capabilities of anarray of coarse-grained reconfigurable (CGR) units enable increasedcompute power that supports both parallel and pipelined computation. ACGR processor can be programmed (e.g., through configuration with aconfiguration file that has been distilled from a high-level program) tosimultaneously execute multiple independent and interdependent data flowgraphs.

FIG. 1 illustrates an example data processing system 100 including a CGRprocessor 110, a host processor 180, and an attached memory 190. Asshown, CGR processor 110 has a coarse-grained reconfigurablearchitecture (CGRA) and includes an array of CGR units 120 such as a CGRarray. CGR processor 110 may include an input-output (I/O) interface 138and a memory interface 139. Array of CGR units 120 may be coupled with(I/O) interface 138 and memory interface 139 via databus 130 which maybe part of a top-level network (TLN). Host processor 180 communicateswith I/O interface 138 via system databus 185, and memory interface 139communicates with memory 190 via memory bus 195.

Array of CGR units 120 may further include compute units and memoryunits that are interconnected with an array-level network (ALN) toprovide the circuitry for execution of a computation graph or a dataflow graph that may have been derived from a high-level program withuser algorithms and functions. The high-level program may include a setof procedures, such as learning or inferencing in an AI or ML system.More specifically, the high-level program may include applications,graphs, application graphs, user applications, computation graphs,control flow graphs, data flow graphs, models, deep learningapplications, deep learning neural networks, programs, program images,jobs, tasks and/or any other procedures and functions that may performserial and/or parallel processing.

In some implementations, execution of the graph(s) may involve usingmore than one CGR processor 110. In some implementations, CGR processor110 may include one or more arrays of CGR units 120.

Host processor 180 may be, or include, a computer such as furtherdescribed with reference to FIG. 2 . Host processor 180 runs runtimeprocesses or implements runtime logic, as further referenced herein.Therefore, host processor 180 or portions of host processor 180 aresometimes also referred to as a runtime processor. In someimplementations, host processor 180 may also be used to run computerprograms, such as the compiler further described herein with referenceto FIG. 6 . In some implementations, the compiler may run on a computerthat is similar to the computer described with reference to FIG. 2 , butseparate from host processor 180.

CGR processor 110 may accomplish computational tasks by executing aconfiguration file (e.g., a processor-executable format (PEF) file). Forthe purposes of this description, a configuration file corresponds to adata flow graph, or a translation of a data flow graph, and may furtherinclude initialization data. A compiler compiles the high-level programto provide the configuration file. In some implementations describedherein, a CGR array 120 is configured by programming one or moreconfiguration stores with all or parts of the configuration file.Therefore, the configuration file is sometimes also referred to as aprogramming file.

A single configuration store may be at the level of the CGR processor110 or the CGR array 120, or a CGR unit may include an individualconfiguration store. The configuration file may include configurationdata for the CGR array and CGR units in the CGR array, and link thecomputation graph to the CGR array. Execution of the configuration fileby CGR processor 110 causes the CGR array(s) to implement the useralgorithms and functions in the data flow graph.

As an example, the CGR processor 110 may be configured to implementfirst and second operations, a recording unit, and a control unit. Thefirst operation generates an output, whereby the size of the output isunknown during a configuration phase. The second operation receives theoutput of the first operation as an input. The recording unit generatescontrol data that is indicative of the size of the output, and thecontrol unit provides the control data to the second operation such thatthe second operation processes the input based on the control data. Ifdesired, the arrays of CGR units 120 in the CGR processor 110 implementthe first and second operations.

In some implementations, the recording unit may generate the controldata while the first operation generates the output. For example, therecording unit may include a counter, and the counter may count a numberof elements in the output to generate the control data.

CGR processor 110 can be implemented on a single integrated circuit (IC)die or on a multichip module (MCM). An IC can be packaged in a singlechip module or a multichip module. An MCM is an electronic package thatmay comprise multiple IC dies and other devices, assembled into a singlemodule as if it were a single device. The various dies of an MCM may bemounted on a substrate, and the bare dies of the substrate areelectrically coupled to the surface or to each other using for someexamples, wire bonding, tape bonding or flip-chip bonding.

FIG. 2 illustrates an example of a computer 200, including an inputdevice 210, a processor 220, a storage device 230, and an output device240. Although the example computer 200 is drawn with a single processor220, other implementations may have multiple processors. Input device210 may comprise a mouse, a keyboard, a sensor, an input port (e.g., auniversal serial bus (USB) port), and/or any other input device known inthe art. Output device 240 may comprise a monitor, printer, and/or anyother output device known in the art. Illustratively, part or all ofinput device 210 and output device 240 may be combined in a networkinterface, such as a Peripheral Component Interconnect Express (PCIe)interface suitable for communicating with CGR processor 110 of FIG. 1 .

Input device 210 is coupled with processor 220, which is sometimes alsoreferred to as host processor 220) to provide input data. If desired,memory 226 of processor 220 may store the input data. Processor 220 iscoupled with output device 240. In some implementations, memory 226 mayprovide output data to output device 240.

Processor 220 further includes control logic 222 and arithmetic andlogic unit (ALU) 224. Control logic 222 may be operable to controlmemory 226 and ALU 224. If desired, control logic 222 may be operable toreceive program and configuration data from memory 226. Illustratively,control logic 222 may control exchange of data between memory 226 andstorage device 230. Memory 226 may comprise memory with fast access,such as static random-access memory (SRAM). Storage device 230 maycomprise memory with slow access, such as dynamic random-access memory(DRAM), flash memory, magnetic disks, optical disks, and/or any othermemory type known in the art. At least a part of the memory in storagedevice 230 includes a non-transitory computer-readable medium (CRM 235),such as used for storing computer programs. The storage device 230 issometimes also referred to as host memory.

FIG. 3 illustrates example details of a CGR architecture 300 including atop-level network (TLN 330) and two CGR arrays (CGR array 310 and CGRarray 320). A CGR array comprises an array of CGR units (e.g., patternmemory units (PMUs), pattern compute units (PCUs), fused-control memoryunits (FCMUs)) coupled via an array-level network (ALN), e.g., a bussystem. The ALN may be coupled with the TLN 330 through several AddressGeneration and Coalescing Units (AGCUs), and consequently withinput/output (I/O) interface 338 (or any number of interfaces) andmemory interface 339. Other implementations may use different bus orcommunication architectures.

Circuits on the TLN in this example include one or more external I/Ointerfaces, including I/O interface 338 and memory interface 339. Theinterfaces to external devices include circuits for routing data amongcircuits coupled with the TLN 330 and external devices, such ashigh-capacity memory, host processors, other CGR processors, FPGAdevices, and so on, that may be coupled with the interfaces.

As shown in FIG. 3 , each CGR array 310, 320 has four AGCUs (e.g.,MAGCU1, AGCU12, AGCU13, and AGCU14 in CGR array 310). The AGCUsinterface the TLN to the ALNs and route data from the TLN to the ALN orvice versa.

One of the AGCUs in each CGR array in this example is configured to be amaster AGCU (MAGCU), which includes an array configuration load/unloadcontroller for the CGR array. The MAGCU1 includes a configurationload/unload controller for CGR array 310, and MAGCU2 includes aconfiguration load/unload controller for CGR array 320. Someimplementations may include more than one array configurationload/unload controller. In other implementations, an array configurationload/unload controller may be implemented by logic distributed amongmore than one AGCU. In yet other implementations, a configurationload/unload controller can be designed for loading and unloadingconfiguration of more than one CGR array. In further implementations,more than one configuration controller can be designed for configurationof a single CGR array. Also, the configuration load/unload controllercan be implemented in other portions of the system, including as astand-alone circuit on the TLN and the ALN or ALNs.

The TLN 330 may be constructed using top-level switches (e.g., switch311, switch 312, switch 313, switch 314, switch 315, and switch 316). Ifdesired, the top-level switches may be coupled with at least one othertop-level switch. At least some top-level switches may be connected withother circuits on the TLN, including the AGCUs, and external I/Ointerface 338.

Illustratively, the TLN 330 includes links (e.g., L11, L12, L21, L22)coupling the top-level switches. Data may travel in packets between thetop-level switches on the links, and from the switches to the circuitson the network coupled with the switches. For example, switch 311 andswitch 312 are coupled by link L11, switch 314 and switch 315 arecoupled by link L12, switch 311 and switch 314 are coupled by link L13,and switch 312 and switch 313 are coupled by link L21. The links caninclude one or more buses and supporting control lines, including forexample a chunk-wide bus (vector bus). For example, the top-levelnetwork can include data, request and response channels operable incoordination for transfer of data in any manner known in the art.

FIG. 4 illustrates an example CGR array 400, including an array of CGRunits in an ALN. CGR array 400 may include several types of CGR unit401, such as FCMUs, PMUs, PCUs, memory units, and/or compute units. Forexamples of the functions of these types of CGR units, see Prabhakar etal., “Plasticine: A Reconfigurable Architecture for Parallel Patterns”,ISCA 2017, Jun. 24-28, 2017, Toronto, ON, Canada.

Illustratively, each of the CGR units may include a configuration store402 comprising a set of registers or flip-flops storing configurationdata that represents the setup and/or the sequence to run a program, andthat can include the number of nested loops, the limits of each loopiterator, the instructions to be executed for each stage, the source ofoperands, and the network parameters for the input and outputinterfaces. In some implementations, each CGR unit 401 comprises anFCMU. In other implementations, the array comprises both PMUs and PCUs,or memory units and compute units, arranged in a checkerboard pattern.In yet other implementations, CGR units may be arranged in differentpatterns.

The ALN includes switch units 403 (S), and AGCUs (each including twoaddress generators 405 (AG) and a shared coalescing unit 404 (CU)).Switch units 403 are connected among themselves via interconnects 421and to a CGR unit 401 with interconnects 422. Switch units 403 may becoupled with address generators 405 via interconnects 420. In someimplementations, communication channels can be configured as end-to-endconnections.

A configuration file may include configuration data representing aninitial configuration, or starting state, of each of the CGR units 401that execute a high-level program with user algorithms and functions oroperations. Program load is the process of setting up the configurationstores 402 in the CGR array 400 based on the configuration data to allowthe CGR units 401 to execute the high-level program. For example, theCGR units 401 may be configured to implement first and secondoperations. The first operation may generate an output of a size that isunknown during configuration with the configuration data, which issometimes also referred to as the configuration phase. Program load mayalso require loading memory units and/or PMUs.

In some implementations, a runtime processor (e.g., the portions of hostprocessor 180 of FIG. 1 that execute runtime processes) may perform theprogram load.

The ALN includes one or more kinds of physical data buses, for example achunk-level vector bus (e.g., 512 bits of data), a word-level scalar bus(e.g., 32 bits of data), and a control bus. For instance, interconnects421 between two switches may include a vector bus interconnect with abus width of 512 bits, and a scalar bus interconnect with a bus width of32 bits. A control bus can comprise a configurable interconnect thatcarries multiple control bits on signal routes designated byconfiguration bits in the CGR array's configuration file. The controlbus can comprise physical lines separate from the data buses in someimplementations. In other implementations, the control bus can beimplemented using the same physical lines with a separate protocol or ina time-sharing procedure.

Physical data buses may differ in the granularity of data beingtransferred. In one implementation, a vector bus can carry a chunk thatincludes 16 channels of 32-bit floating-point data or 32 channels of16-bit floating-point data (i.e., 512 bits) of data as its payload. Ascalar bus can have a 32-bit payload and carry scalar operands orcontrol information. The control bus can carry control handshakes suchas tokens and other signals.

The vector and scalar buses can be packet-switched, including headersthat indicate a destination of each packet and other information such assequence numbers that can be used to reassemble a file when the packetsare received out of order. Each packet header can contain a destinationidentifier that identifies the geographical coordinates of thedestination switch unit (e.g., the row and column in the array), and aninterface identifier that identifies the interface on the destinationswitch (e.g., North, South, East, West, etc.) used to reach thedestination unit.

A CGR unit 401 may have four ports (as drawn) to interface with switchunits 403, or any other number of ports suitable for an ALN. Each portmay be suitable for receiving and transmitting data, or a port may besuitable for only receiving or only transmitting data.

A switch unit 403, as shown in the example of FIG. 4 , may have eightinterfaces. The North, South, East and West interfaces of a switch unitmay be used for links between switch units 403 using interconnects 421.The Northeast, Southeast, Northwest and Southwest interfaces of a switchunit 403 may each be used to make a link with an FCMU, PCU or PMUinstance 401 using one of the interconnects 422. Two switch units 403 ineach CGR array quadrant have links to an AGCU using interconnects 420.The coalescing unit 404 of the AGCU arbitrates between the AGs 405 andprocesses memory requests. Each of the eight interfaces of a switch unit403 can include a vector interface, a scalar interface, and a controlinterface to communicate with the vector network, the scalar network,and the control network. In other implementations, a switch unit 403 mayhave any number of interfaces.

During execution of a graph or subgraph in a CGR array 400 afterconfiguration, data can be sent via one or more switch units 403 and oneor more links 421 between the switch units to the CGR units 401 usingthe vector bus and vector interface(s) of the one or more switch units403 on the ALN. A CGR array may comprise at least a part of CGR array400, and any number of other CGR arrays coupled with CGR array 400.

A data processing operation implemented by CGR array configuration maycomprise multiple graphs or subgraphs specifying data processingoperations that are distributed among and executed by corresponding CGRunits (e.g., FCMUs, PMUs, PCUs, AGs, and CUs).

FIG. 5 illustrates an example 500 of a PMU 510 and a PCU 520, which maybe combined in an FCMU 530. PMU 510 may be directly coupled to PCU 520,or optionally via one or more switches. PMU 510 includes a scratchpadmemory 515, which may receive external data, memory addresses, andmemory control information (e.g., write enable, read enable) via one ormore buses included in the ALN. PCU 520 includes two or more processorstages, such as SIMD 521 through SIMD 526, and configuration store 528.The processor stages may include ALUs, or SIMDs, as drawn, or any otherreconfigurable stages that can process data.

Each stage in PCU 520 may also hold one or more registers (not drawn)for short-term storage of parameters. Short-term storage, for exampleduring one to several clock cycles or unit delays, allows forsynchronization of data in the PCU pipeline.

FIG. 6 shows a compute environment 600 that provides on-demand networkaccess to a pool of reconfigurable data flow resources 678 that can berapidly provisioned and released with minimal management effort orservice provider interaction. The pool of reconfigurable data flowresources 678 includes memory (e.g., attached memory 190 of FIG. 1 ),arrays of CGR units, and busses (e.g., memory bus 195 of FIG. 1 and/orTLN 330 of FIG. 3 ) that couple the arrays of CGR units and the memory.

The busses or transfer resources enable the arrays of CGR units toreceive and send data. Examples of the busses include peripheralcomponent interface express (PCIe) channels, direct memory access (DMA)channels, double data-rate (DDR) channels, Ethernet channels, andInfiniBand channels. In some implementations, the busses include atleast one of a DMA channel, a DDR channel, a PCIe channel, an Ethernetchannel, or an InfiniBand channel.

The arrays of CGR units (e.g., arrays of compute units and memory units)are arranged in one or more reconfigurable processors (e.g., CGRprocessor 110 of FIG. 1 ) and may be coupled with each other in aprogrammable interconnect fabric (e.g., ALN 120 of FIG. 1 ). In someimplementations, the arrays of CGR units are aggregated as a uniformpool of resources that are assigned to the execution of userapplications.

The memory of the pool of reconfigurable data flow resources 678 may beusable by the arrays of CGR units to store data. Examples of the memoryinclude main memory (e.g., off-chip/external dynamic random-accessmemory (DRAM)) and/or local secondary storage (e.g., local disks (e.g.,hard disk drive (HDD), solid-state drive (SSD))). The memory units ofthe arrays of CGR units may include PMUs, latches, registers, and/orcaches (e.g., SRAM).

The pool of reconfigurable data flow resources 678 is dynamicallyscalable to meet the performance objectives of applications 602 (or userapplications 602). In some implementations, the applications 602 accessthe pool of reconfigurable data flow resources 678 over one or morenetworks (e.g., Internet).

The pool of reconfigurable data flow resources 678 may have differentcompute scales and hierarchies according to different implementations ofthe technology disclosed.

In one example, the pool of reconfigurable data flow resources 678 is anode (or a single machine) with arrays of CGR units that are arranged ina plurality of reconfigurable processors, supported by bus and memory.The node also includes a host processor (e.g., a CPU) that exchangesdata with the plurality of reconfigurable processors, for example, overa PCIe interface. The host processor includes a runtime processor thatmanages resource allocation, memory mapping, and execution of theconfiguration files for applications requesting execution from the hostprocessor.

In another example, the pool of reconfigurable data flow resources 678is a rack (or cluster) of nodes, such that each node in the rack runs arespective plurality of reconfigurable processors, and includes arespective host processor configured with a respective runtimeprocessor. The runtime processors are distributed across the nodes andcommunicate with each other so that they have unified access to thereconfigurable processors attached not just to their own node on whichthey run, but also to the reconfigurable processors attached to everyother node in the data center.

The nodes in the rack are connected, for example, over Ethernet orInfiniBand (IB). In yet another example, the pool of reconfigurable dataflow resources 678 is a pod that comprises a plurality of racks. In yetanother example, the pool of reconfigurable data flow resources 678 is asuperpod that comprises a plurality of pods. In yet another example, thepool of reconfigurable data flow resources 678 is a zone that comprisesa plurality of superpods. In yet another example, the pool ofreconfigurable data flow resources 678 is a data center that comprises aplurality of zones.

Users may execute applications 602 on the compute environment 600.Therefore, applications 602 are sometimes also referred to as userapplications. The applications 602 are executed on the pool ofreconfigurable data flow resources 678 in a distributed fashion byprogramming the individual compute and memory components toasynchronously receive, process, and send data and control information.

In the pool of reconfigurable data flow resources 678, computation canbe executed as deep, nested data flow pipelines that exploit nestedparallelism and data locality very efficiently. These data flowpipelines contain several stages of computation, where each stage readsdata from one or more input buffers with an irregular memory accesspattern, performs computations on the data while using one or moreinternal buffers or scratchpad memory to store and retrieve intermediateresults, and produce outputs that are written to one or more outputbuffers. The structure of these pipelines depends on the control anddata flow graph representing the application. Pipelines can bearbitrarily nested and looped within each other.

The applications 602 comprise high-level programs. A high-level programmay include source code written in programming languages like C, C++,Java, JavaScript, Python, and/or Spatial, for example, using deeplearning frameworks 614 such as PyTorch, TensorFlow, ONNX, Caffe, and/orKeras. The high-level program can implement computing structures andalgorithms of machine learning models like AlexNet, VGG Net, GoogleNet,ResNet, ResNeXt, RCNN, YOLO, SqueezeNet, SegNet, GAN, BERT, ELMo, USE,Transformer, and/or Transformer-XL.

Software development kit (SDK) 642 generates computation graphs (e.g.,data flow graphs, control graphs) 636 of the high-level programs of theapplications 602. The SDK 642 transforms the input behavioraldescription of the high-level programs into an intermediaterepresentation such as the computation graphs 636. This may include codeoptimization steps like false data dependency elimination, dead-codeelimination, and constant folding. The computation graphs 636 encode thedata and control dependencies of the high-level programs.

The computation graphs 636 comprise nodes and edges. The nodes canrepresent compute operations and memory allocations. The edges canrepresent data flow and flow control. In some implementations, each loopin the high-level programs can be represented as a “controller” in thecomputation graphs 636. The computation graphs 636 support branches,loops, function calls, and other variations of control dependencies. Insome implementations, after the computation graphs 636 are generated,additional analyses or optimizations focused on loop transformations canbe performed, such as loop unrolling, loop pipelining, loopfission/fusion, and loop tiling.

The SDK 642 also supports programming the reconfigurable data flowresources in the pool of reconfigurable data flow resources 678 atmultiple levels, for example, from the high-level deep learningframeworks 614 to C++ and assembly language. In some implementations,the SDK 642 allows programmers to develop code that runs directly on thereconfigurable data flow resources. In other implementations, the SDK642 provides libraries that contain predefined functions like linearalgebra operations, element-wise tensor operations, non-linearities, andreductions that are used for creating, executing, and profiling thecomputation graphs 636 on the reconfigurable data flow resources. TheSDK 642 communicates with the deep learning frameworks 614 viaApplication Programming Interfaces (APIs) 624.

A compiler 648 transforms the computation graphs 636 into ahardware-specific configuration, which is specified in an execution file656 generated by the compiler 648. In one implementation, the compiler648 partitions the computation graphs 636 into memory allocations andexecution fragments, and these partitions are specified in the executionfile 656. Execution fragments represent operations on data. An executionfragment can comprise portions of a program representing an amount ofwork. An execution fragment can comprise computations encompassed by aset of loops, a set of graph nodes, or some other unit of work thatrequires synchronization. An execution fragment can comprise a fixed orvariable amount of work, as intended by the program. Different ones ofthe execution fragments can contain different amounts of computation.Execution fragments can represent parallel patterns or portions ofparallel patterns and are executable asynchronously.

In some implementations, the partitioning of the computation graphs 636into the execution fragments includes treating calculations within atleast one innermost loop of a nested loop of the computation graphs 636as a separate execution fragment. In other implementations, thepartitioning of the computation graphs 636 into the execution fragmentsincludes treating calculations of an outer loop around the innermostloop of the computation graphs 636 as a separate execution fragment. Inthe case of imperfectly nested loops, operations within a loop body upto the beginning of a nested loop within that loop body are groupedtogether as a separate execution fragment.

Memory allocations represent the creation of logical memory spaces inon-chip and/or off-chip memories for data used to implement thecomputation graphs 636, and these memory allocations are specified inthe execution file 656. Memory allocations define the type and thenumber of hardware resources (functional units, storage, or connectivitycomponents). Main memory (e.g., DRAM) is memory outside thereconfigurable processors for which the memory allocations can be made.Scratchpad memory (e.g., SRAM) is memory inside the reconfigurableprocessors for which the memory allocations can be made. Other memorytypes for which the memory allocations can be made for various accesspatterns and layouts include read-only lookup-tables (LUTs), fixed sizequeues (e.g., FIFOs), and register files.

The compiler 648 binds memory allocations to virtual memory units andbinds execution fragments to virtual compute units, and these bindingsare specified in the execution file 656. In some implementations, thecompiler 648 partitions execution fragments into memory fragments andcompute fragments, and these partitions are specified in the executionfile 656.

The compiler 648 assigns the memory fragments to the virtual memoryunits and assigns the compute fragments to the virtual compute units,and these assignments are specified in the execution file 656. Eachmemory fragment is mapped operation-wise to the virtual memory unitcorresponding to the memory being accessed. Each operation is lowered toits corresponding configuration intermediate representation for thatvirtual memory unit. Each compute fragment is mapped operation-wise to anewly allocated virtual compute unit. Each operation is lowered to itscorresponding configuration intermediate representation for that virtualcompute unit.

The compiler 648 allocates the virtual memory units to physical memoryunits of a reconfigurable processor (e.g., pattern memory units (PMUs)of the reconfigurable processor) and allocates the virtual compute unitsto physical compute units of the reconfigurable processor (e.g., patterncompute units (PCUs) of the reconfigurable processor), and theseallocations are specified in the execution file 656. The compiler 648places the physical memory units and the physical compute units ontopositions in the arrays of CGR units of the pool of reconfigurable dataflow resources and routes data and control networks between the placedpositions, and these placements and routes are specified in theexecution file 656.

The compiler 648 may translate the applications 602 developed withcommonly used open-source packages such as Keras and/or PyTorch intoreconfigurable processor specifications. The compiler 648 generates theconfiguration files with configuration data for the placed positions andthe routed data and control networks. In one implementation, thisincludes assigning coordinates and communication resources of thephysical memory and compute units by placing and routing units onto thearrays of the CGR units while maximizing bandwidth and minimizinglatency.

A runtime processor 666 receives the execution file 656 from the SDK 642and uses the execution file 656 for resource allocation, memory mapping,and execution of the configuration files for the applications 602 on thepool of reconfigurable data flow resources 678. The runtime processor666 may communicate with the SDK 642 over APIs 654 (e.g., Python APIs).If desired, the runtime processor 666 can directly communicate with thedeep learning frameworks 614 over APIs 652 (e.g., C/C++ APIs).

The runtime processor 666 may be operatively coupled to the pool ofreconfigurable data flow resources 678 via a local bus 672. If desired,the local bus 672 may be a PCIe bus or any other local bus that enablesthe runtime processor 666 to exchange data with the pool ofreconfigurable data flow resources 678.

The runtime processor 666 parses the execution file 656, which includesa plurality of configuration files. Configuration files in the pluralityof configurations files include configurations of the virtual data flowresources that are used to execute the user applications 602. Theruntime processor 666 allocates a subset of the arrays of CGR units inthe pool of reconfigurable data flow resources 678 to the virtual dataflow resources.

The runtime processor 666 then loads the configuration files for theapplications 602 to the subset of the arrays of CGR units. In thescenario in which the execution file 656 includes two user applications602 (e.g., a first and a second user application), the runtime processor666 is configured to load a first configuration file for executing thefirst user application to a first subset of the arrays of CGR units inthe pool of reconfigurable data flow resources 678, and to load a secondconfiguration file for executing the second user application to a secondsubset of the arrays of CGR units in the pool of reconfigurable dataflow resources 678. In some implementations, the memory and the arraysof CGR units of the one or more reconfigurable processors in the pool ofreconfigurable data flow resources 678 are aggregated as a uniform poolof resources that are assigned to the execution of the first and seconduser applications 602. The runtime processor 666 then starts executionof the user applications 602 on the subsets of the arrays of CGR units.

An application for the purposes of this description includes theconfiguration files for reconfigurable data flow resources in the poolof reconfigurable data flow resources 678 compiled to execute a missionfunction procedure or set of procedures such as inferencing or learningin an artificial intelligence or machine learning system. A virtualmachine for the purposes of this description comprises a set ofreconfigurable data flow resources (including arrays of CGR units in oneor more reconfigurable processor, bus, and memory) configured to supportexecution of an application in arrays of CGR units and associated busand memory in a manner that appears to the application as if there werea physical constraint on the resources available, such as would beexperienced in a physical machine. The virtual machine can beestablished as a part of the application of the mission function thatuses the virtual machine, or it can be established using a separateconfiguration mechanism. In implementations described herein, virtualmachines are implemented using resources of the pool of reconfigurabledata flow resources 678 that are also used in the application, and sothe configuration files for the application include the configurationdata for its corresponding virtual machine, and links the application toa particular set of CGR units in the arrays of CGR units and associatedbus and memory.

The runtime processor 666 implements an application in a virtual machinethat is allocated a particular set of reconfigurable data flowresources. The virtual machine includes a particular set of CGR units,which can include some or all CGR units of a single reconfigurableprocessor or of multiple reconfigurable processors, along withassociated bus and memory (e.g., PCIe channels, DMA channels, DDRchannels, DRAM memory).

The runtime processor 666 respects the topology information (e.g.,topology information 704 of FIG. 7 ) in the execution file 656 whenallocating CGR units to the virtual data flow resources requested in theexecution file 656.

The input and output memories as well as the compute units areconfigured for each operation at compile time. Thus, the sizes of thevector data of input and output of an operation is expected to be knownat compile time.

However, as mentioned above, some functions or operations produce anoutput of a size that is unknown at compile time. As an example,consider the operation nonzero( ). In one implementation, the operationnonzero( ) receives an input tensor and returns a two-dimensional outputtensor in which each row is the index for a non-zero value in the inputtensor. Thus, if the input tensor has N dimensions and Z non-zeroelements, the output tensor is of size (Z×N).

FIG. 7 is a diagram of an illustrative operation that generates anoutput of a size that is unknown at compile time. As shown in FIG. 7 ,the operation is a filter operation 710 that receives an input tensor720 and generates an output tensor 740 containing elements that satisfya predicate 730. In the scenario in which the input tensor 720 includesN elements, the filter operation 710 may produce an output tensor 740that contains between O and N elements.

It should be noted that the terms “dynamically-sized data structure” and“tensor” or “output tensor” as well as the terms “operator that producesthe dynamically-sized data structure” and “filter” are hereinafter usedinterchangeably for simplifying the discussion. However, one skilled inthe art would appreciate that the described technology equally appliesto operators other than filters which produce dynamically-sized datastructures.

FIG. 8 is a diagram of an illustrative data processing system 800 with areconfigurable processor 805 that is coupled to external memory 870. Insome implementations, reconfigurable processor 805 is a CGR processor(e.g., CGR processor 110 of FIG. 1 ) that includes arrays of CGR units(e.g., CGR arrays 120 of FIG. 1 ) having memory units and compute units.

As shown in FIG. 8 , the reconfigurable processor 805 is configured toimplement an application. The application may include a first operation810, a second operation 820, a recording unit 830, and a control unit840.

Illustratively, a compiler (e.g., the compiler of FIG. 1 ) may generateconfiguration data for configuring the reconfigurable processor toimplement the application including the first operation 810, the secondoperation 820, the recording unit 830, and the control unit 840.

Runtime logic (e.g., runtime logic operating on host processor 180 ofFIG. 1 ) may be configured to program the reconfigurable processor 805with the configuration data such that the reconfigurable processor 805implements the application during a configuration phase. For example,the runtime logic may configure one or more compute units and/or memoryunits of the CGR arrays of the reconfigurable processor 805 to implementfirst and second operations 810, 820, recording unit 830, and controlunit 840.

The first operation 810 may generate an output 852, and the secondoperation 820 may receive the output 852 as an input 854. In someimplementations, the data of output 852 may be stored in one or morememory units 850, and the second operation 820 may read the data fromthe one or more memory units 850 as input 854.

The size of the output 852 may be unknown at compile time, and thusduring the configuration phase during which the host processor (e.g.,the runtime logic of host processor 180 of FIG. 1 ) configures thereconfigurable processor 805. For example, the first operation 810 maybe a nonzero( ) or a filter operation as described with reference toFIG. 7 .

In the scenario in which the first operation 810 (e.g., a filteroperation) receives an input tensor having a predetermined number ofelements (e.g., N elements), the output 852 of the first operation 810may include a number of elements M that is smaller than or equal to apredetermined maximum number of elements, whereby the predeterminedmaximum number of elements is equal to the predetermined number ofelements of the input tensor (i.e., M≤N). In this scenario, the compilermay generate in the configuration data a first connection for the output852 and a second connection for the input 854, and each one of the firstand second connections is suitable for a transmission of thepredetermined maximum number of elements (i.e., N elements).

For storing the output 852 in memory 850, the compiler may allocate asmuch memory space (e.g., as many memory units 850) for the output 852and for any subsequent memories downstream of the first operation 810that are affected by the dynamically-sized output 852 as memory space isrequired for storing the input of the first operation 810. For routingthe output 852 to any subsequent operations, the compiler may allocateinterconnections that are able to transfer the same amount of data asthe interconnection that conveys input data to the first operation 810.

If desired, the compiler may add control circuitry (e.g., recording unit830 and control unit 840) to the first operation that produces thedynamically-sized output 852 at compile time. The reconfigurableprocessor 805 may be configured with the control circuitry for assistingwith the execution of the application that includes the first operation810 that produces dynamically-sized output 852.

The control circuitry may improve the efficiency of the execution ofoperations such as the second operation 820 that are downstream from thefirst operation that produces the dynamically-sized output 852.Moreover, the control circuitry may prevent that garbage data is readout of the one or more memory units 850 that store the dynamically-sizedoutput 852 due to the dynamically-sized data of the output 852potentially not writing to the entire memory space of the one or morememory units 850 that were allocated during compilation time.

Illustratively, the recording unit 830 may generate control data that isindicative of the size of the output 852 of the first operation 810. Forexample, the control data may be a scalar that includes the number ofelements in the output 852.

In some implementations, the recording unit 830 may generate the controldata while the first operation 810 generates the output. For example,the recording unit 830 may include a counter. The counter may count thenumber of elements in the output 852 to generate the control data. Forexample, the counter may access the first operation 810 and incrementconditionally based on a predicate of the first operations 810 (e.g.,when a filter adds a new element to the output 852). If desired, thecontrol data may be a tuple. For example, the control data may includethe number of elements and the size of each element of output 852.

In other implementations, the recording unit 830 may generate thecontrol data by accessing the one or more memory units 850 to determinethe size of the output 852 of the first operation 810.

Illustratively, the control unit 840 may fetch the control data from therecording unit 830. For example, the control unit 840 may fetch thecontrol data from the recording unit 830 when the operator has finishedexecuting the operation. In some implementations, the control mechanismmay read out the value of the counter.

The control unit 840 may send the control data out onto the data network(e.g., the ALN of FIG. 1 ) to which the operations such as the secondoperation 820 that are consuming the output 852 of the first operation810 (i.e., the consumers) are connected. As an example, the control unit840 may send the control data to the second operation 820 overconnection 872. As another example, the control unit 840 may store thecontrol data in external memory 870 that is coupled to thereconfigurable processor 805 over connection 873, and the consumersincluding the second operation 820 may retrieve the control data fromthe external memory 870 over connection 874.

Illustratively, the reconfigurable processor 805 is configured toimplement a synchronization unit 860. The synchronization unit 860 mayinform the consumer of a dynamically-sized output when the producer ofthe dynamically-sized output has finished the operation. Thereby, thesynchronization unit 860 may ensure that the consumer is prevented fromusing stale stored data. For example, the synchronization unit 860 mayinform the second operation 820 when the first operation 810 hasgenerated the output 852.

In some implementations, the data flow architectures of reconfigurableprocessor 805 may be heavily pipelined. For example, a compute unit thatimplements the first function 810 may receive an input from a memoryunit and write the output 852 to another memory unit 850.

For example, the reconfigurable processor 805 may be configured to storethe output 852 in a buffer (e.g., memory unit 850) during a writeoperation, and the control unit 840 may direct the second operation 820during a read operation following the write operation to read data asthe input 854 from the buffer 850 that was stored during the writeoperation.

In some scenarios, the memory unit 850 may operate as a multi-bufferedmeta-pipeline that allows readers (i.e., the second operation 820) andwriters (i.e., the first operation 810) to operate on independentbatches of the data. FIG. 9A is a diagram of an illustrative buffer 900.The address space of buffer 900 may be partitioned such that a writeoperation is performed at a first portion 920 of the buffer 900 and aread operation at a second portion 910 of the buffer 900. Such a buffer900 is sometimes also referred to as a double buffer.

For example, the reconfigurable processor 805 of FIG. 8 may beconfigured to enable the first operation 810 to write a first portion ofthe output 952 to a first portion 920 of the buffer 900, while thesecond operation 820 reads a first portion of the input 951 that isdifferent than the first portion of the output 952 from a second portion910 of the buffer 900 that is different than the first portion 920 ofthe buffer 900.

When the first operation (e.g., operation 810 of FIG. 8 ) has finishedwriting the first portion 952 of the output and the second operation(e.g., operation 820 of FIG. 8 ) has finished reading the first portion951 of the input, the addresses of the first and second portions mayswap.

Thus, the reconfigurable processor (e.g., reconfigurable processor 805of FIG. 8 ) is configured to enable the first operation (e.g., operation810 of FIG. 8 ) to write a second portion 953 of the output to thesecond portion 910 of the buffer 900 (e.g., memory units 850 of FIG. 8), while the second operation (e.g., operation 820 of FIG. 8 ) reads thefirst portion 952 of the output from the first portion 920 of the buffer900 as a second portion of the input. This is illustratively shown inFIG. 9B.

FIG. 10 is a diagram of illustrative dependencies between read and writeoperations in a reconfigurable processor that is configured to implementa function that generates a dynamically-sized output as shown in FIG. 8.

A first operation may generate a dynamically-sized output that iswritten to a memory unit during a write operation 1010. If desired, arecording unit may write control data that is indicative of the size ofthe dynamically-sized output of the first operation to the memory unitat the end of the first operation.

For example, the first function 810 of FIG. 8 may generatedynamically-sized output 852 that is written to memory unit 850 during awrite operation. If desired, recording unit 830 may write control datathat is indicative of the size of the output 852 of the first operation810 to the memory unit 850 at the end of the write operation.

During read operation 1020, reader R0 may read the contents of thecontrol data (e.g., a scalar). In some implementations, reader R0 maypush the control data into the data network. In other implementations,reader R0 may write the control data to external storage. For example,the control unit 840 of FIG. 8 may retrieve the control data from therecording unit 830 and send the control data directly to the secondoperation 820 over connection 872, or the control data may write thecontrol data over connection 873 to the external memory 870 from whichthe second operation 820 may retrieve the control data over connection874.

Once reader R0 has finished reading the contents of the control data andinformed downstream operations, reader R0 may hand the control toanother reader R1. Reader R1 actually performs the data read from thememory unit during operation 1030. For example, control unit 840 of FIG.8 may hand the control to second operation 820, and the second operation820 may perform the data read from the memory unit 850.

Illustratively, the filter operation shown in FIG. 7 may be implementedtogether with control circuitry on a reconfigurable processor as shownin FIG. 11 . As shown in FIG. 11 , the filter 710 of FIG. 7 may beimplemented in compute unit 1120 of the reconfigurable processor, whilethe input tensor 720 is stored in memory unit 1110 and output tensor 740is stored in memory unit 1130 of the reconfigurable processor. In someimplementations, the reconfigurable processor may be configured toimplement control circuitry (e.g., a recording unit that generatescontrol data that is indicative of the size of the output tensor that isstored in output memory unit 1130 and a control unit that communicatesthe control data to compute units downstream of the output memory unit1130).

In some implementations, the recording unit may be implemented usingcounters. For example, consider the scenario in which the input memoryunit 1110 stores a statically-sized tensor of a predetermined size. Inthis scenario, the output memory unit 1130 may maintain a counter with amaximum value that is set to the predetermined size of the input tensor.The counter stride may be set by an external scalar port that receivesthe filter predicate (e.g., filter predicate 730 of FIG. 7 ). When thecounter reaches the maximum value, a context change occurs that readsthe current counter value and uses the current counter value to set themaximum value of the read context.

For example, compute unit 1120 may retrieve input data 1115 from inputmemory unit 1110. During a write operation 1140, compute unit 1120 maywrite data 1125 to output memory unit 1130. Simultaneously, therecording unit may store a current counter value in the output memoryunit 1130. When the write operation 1140 finishes, compute unit 1120and/or output memory unit 1130 may issue a WRITE_DONE signal 1145.

Upon issuance of the WRITE_DONE signal 1145, a control unit may performa read operation READ1 1150 to retrieve the current counter value fromthe output memory unit 1130 and transmit the current counter value to acompute unit downstream of the output memory unit 1130 for readoperation READ0 1160. The read operation READ1 1150 may issue aREAD1_DONE signal 1155 when the current counter value has beentransmitted to compute unit downstream of the output memory unit 1130.

Upon issuance of the READ1_DONE signal 1155, the compute unit that isdownstream of the output memory unit 1130 may, during read operationREAD0 1160, retrieve the content of the output memory unit 1130 usingthe current counter value. When the read operation READ0 1160 hasfinished, the read operation READ0 1160 may issue a READ0_DONE signal1165 so that another write operation WRITE 1140 may start.

Thus, read operation READ1 1150 is blocked by write operation WRITE1140, which in turn blocks read operation READ0 1160. Thereby, thecontrol circuitry ensures that read operation READ0 1160 uses the actualoutput size of the output tensor in output memory unit 1130.

Illustratively, the size of output data 1125 (i.e., the current countervalue) at the end of the write operation performed by compute unit 1120may be communicated to all downstream operations. The downstreamoperations may use the current counter value to set the maximum valuesof their counters in the same way as the filter operation does. Ifdesired, the output tensor size may be stored in external memory (e.g.,DRAM) such as external memory 870 of FIG. 8 .

FIG. 12 is a diagram of an illustrative implementation of anillustrative group-by operation and associated control circuitry on areconfigurable processor. A group-by operation may receive an inputtensor and generate an output tensor with histogram values representinga frequency of each bin value in the input tensor.

Compute unit 1220 of the reconfigurable processor may implement thegroup-by operation, while the input tensor 1215 is stored in memory unit1210, and the output tensor 1225 is stored in memory unit 1230 of thereconfigurable processor. In some implementations, the reconfigurableprocessor may be configured to implement control circuitry (e.g., arecording unit that generates control data that is indicative of thesize of the output tensor that is stored in output memory unit 1230 anda control unit that communicates the control data to consumersdownstream of the output memory unit 1230).

Illustratively, the space allocated for the output tensor 1225 in theoutput memory unit 1230 is determined by the upper bound of the inputtensor length. The initial number of bins (i.e., the histogram length)may be set to one in the output memory unit 1230. In someimplementations, the recording unit may be implemented as a memoryelement in output memory unit 1230 that stores the current number ofbins.

The compute unit 1220 may receive the input tensor 1215 from inputmemory unit 1210 and, for each element in the input tensor 1215,increment the value in the appropriate bin number in the output tensor1225. If desired, the compute unit 1220 may check whether the bin valueis greater than the current number of bins, update the current number ofbins if needed (WRITE0 operation 1270), and communicate the updatedcurrent number of bins to the output memory unit 1230.

When the compute unit 1220 has finished the group-by operation on theinput tensor 1215, the output memory unit 1230 may issue a WRITE0_DONEsignal 1275. Upon issuance of the WRITE0_DONE signal 1275, the computeunit 1220 may write the vector bin values 1225 to the output memory unit1230 (WRITE1 operation 1240). When the WRITE1 operation 1240 finishes,compute unit 1220 and/or output memory unit 1230 may issue a WRITE1_DONEsignal 1245.

Upon issuance of the WRITE1_DONE signal 1245, the compute unit 1220 mayretrieve the old bin value 1235 during read operations READ1 1250. Theread operation READ1 1250 may issue a READ1_DONE signal 1255 when theold bin value 1235 has been transmitted to the compute unit 1220.

Upon issuance of the READ1_DONE signal 1255, the consumers downstream ofthe output memory unit 1230 may, during read operation READ0 1260,retrieve the current number of bins and the content from the outputmemory unit 1230 using the current number of bins. When the readoperation READ0 1260 has finished, another write operation WRITE0 1270may start.

Thus, read operation READ0 1260 is blocked by read operation READ1 1250,which in turn is blocked by write operation WRITE1 1240, which isblocked by write operation WRITE0 1270. Thereby, the control circuitryensures that read operation READ0 uses the actual output size of theoutput tensor in output memory unit 1230.

Illustratively, the size of output tensor 1225 (i.e., the current numberof bins) at the end of the write operation performed by compute unit1220 may be communicated to all downstream operations.

FIG. 13 is a diagram of an illustrative implementation of anillustrative unique operation and associated control circuitry on areconfigurable processor. A unique operation may receive an input tensorand generate an output tensor with no duplicates.

Illustratively, the unique operation may be implemented together withcontrol circuitry on a reconfigurable processor as shown in FIG. 13 . Asshown in FIG. 13 , the unique operation may be implemented in computeunit 1320 of the reconfigurable processor, while the input tensor isstored in memory unit 1310 and the output tensor is stored in memoryunit 1330 of the reconfigurable processor. In some implementations, thereconfigurable processor may be configured to implement controlcircuitry (e.g., a recording unit that generates control data that isindicative of the size of the output tensor that is stored in outputmemory unit 1330 and a control unit that communicates the control datato compute units downstream of the output memory unit 1330).

In some implementations, the recording unit may be implemented usingcounters. For example, consider the scenario in which the input memoryunit 1310 stores a statically-sized tensor of a predetermined size. Inthis scenario, the output memory unit 1330 may maintain a counter with amaximum value that is set to the predetermined size of the input tensor.For example, the output memory unit 1330 may retrieve the predeterminedsize of the input tensor over connection 1335 from the input memory unit1310. The counter stride may be set by an external scalar provided bythe compute unit 1320.

For example, compute unit 1320 may retrieve input data 1315 from inputmemory unit 1310. During a write operation 1340, compute unit 1320 maywrite data 1325 to output memory unit 1330. Simultaneously, therecording unit may store a current counter value in the output memoryunit 1330. When the write operation 1340 finishes, compute unit 1320and/or output memory unit 1330 may issue a WRITE_DONE signal 1345.

Upon issuance of the WRITE_DONE signal 1345, a control unit may performa read operation READ1 1350 to retrieve the current counter value fromthe output memory unit 1330 and transmit the current counter value to acompute unit downstream of the output memory unit 1330 for readoperation READ0 1360. The read operation READ1 1350 may issue aREAD1_DONE signal 1355 when the current counter value has beentransmitted to compute unit downstream of the output memory unit 1330.

Upon issuance of the READ1_DONE signal 1355, the compute unit that isdownstream of the output memory unit 1330 may, during read operationREAD0 1360, retrieve the content of the output memory unit 1330 usingthe current counter value. When the read operation READ0 1360 hasfinished, the read operation READ0 1360 may issue a READ0_DONE signal sothat another write operation WRITE 1340 may start.

Thus, read operation READ1 1350 is blocked by write operation WRITE1340, and READ1 1350 blocks read operation READ0 1360. Thereby, thecontrol circuitry ensures that read operation READ0 uses the actualoutput size of the output tensor in output memory unit 1330.

Illustratively, the size of output data 1325 (i.e., the current countervalue) at the end of the write operation performed by compute unit 1320may be communicated to all downstream operations.

The illustrative operations shown in FIGS. 11, 12, and 13 areparallelizable across multiple compute unit and memory unit streams totrade-off resource usage with throughput. Thereby, each computeunit/memory unit stream of the compute unit and memory unit streams maytrack the size of its corresponding dynamically-sized output data slice.Furthermore, the control circuitry may be extended to provide supportfor higher dimensional dynamically-sized output data and enabledynamically tracking each data-dependent loop-bound within the loopnest.

FIG. 14 is a flowchart showing illustrative operations 1400 that runtimelogic of a host processor may perform for operating a data processingsystem with a reconfigurable processor. For example, runtime logic ofhost processor 180 of FIG. 1 may perform operations for operating dataprocessing system 100 with CGR processor 110.

During operation 1410, the runtime logic may configure thereconfigurable processor such that the reconfigurable processorimplements a first operation that generates an output, wherein a size ofthe output is unknown during a configuration phase, a second operationthat receives the output of the first operation as an input, a recordingunit that generates control data that is indicative of the size of theoutput, and a control unit that provides the control data to the secondoperation, wherein the second operation processes the input based on thecontrol data.

For example, the runtime logic may configure the reconfigurableprocessor 805 of FIG. 8 such that the reconfigurable processor 805implements a first operation 810, a second operation 820, a recordingunit 830, and a control unit 840. The first operation may generate anoutput 852, whereby a size of the output 852 is unknown during aconfiguration phase. The second operation 820 may receive the output 852of the first operation 810 as an input 854. The recording unit maygenerate control data that is indicative of the size of the output 852,and the control unit 840 may provide the control data to the secondoperation 820, whereby the second operation processes the input 854based on the control data.

If desired, the runtime logic may configure the reconfigurable processorsuch that the reconfigurable processor implements a synchronization unitthat informs the second operation when the first operation has generatedthe output. For example, the runtime logic may configure thereconfigurable processor 805 of FIG. 8 such that the reconfigurableprocessor 805 implements a synchronization unit 860 that informs thesecond operation 820 when the first operation 810 has generated theoutput 852.

In some implementations, the data processing system may include acompiler (e.g., compiler of host processor 180 of FIG. 1 ). The compilermay generate configuration data for configuring the reconfigurableprocessor to implement the first operation, the second operation, therecording unit, and the control unit. For example, the compiler maygenerate configuration data for configuring the reconfigurable processor805 of FIG. 8 to implement the first operation 810, the second operation820, the recording unit 830, and the control unit 840.

Illustratively, the output has a number of elements that is smaller thanor equal to a predetermined maximum number of elements, and the compilermay generate in the configuration data a first connection for the outputand a second connection for the input, whereby each one of the first andsecond connections is able to transport the predetermined maximum numberof elements. For example, the compiler may generate in the configurationdata a first connection for the output 852 between the compute unit thatimplements the first operation of FIG. 8 and the memory unit 850 and asecond connection for the input 854 of the second operation, wherebyeach one of the first and second connections is able to transport thepredetermined maximum number of elements.

In some implementations, the data processing system may use runtimelogic to program the reconfigurable processor with configuration data.

If desired, a non-transitory computer-readable storage medium includesinstructions that, when executed by a processing unit (e.g., runtimelogic of host processor 180 of FIG. 1 ), cause the processing unit tooperate a system (e.g., data processing system 100 of FIG. 1 ) byperforming operation 1410 of FIG. 14 .

For example, a non-transitory computer-readable storage medium includesinstructions that, when executed by a processing unit, cause theprocessing unit to operate a data processing system that comprises areconfigurable processor. The instructions include configuring thereconfigurable processor with configuration data such that thereconfigurable processor implements a first operation that generates anoutput, wherein a size of the output is unknown, a second operation thatreceives the output of the first operation as an input, a recording unitthat generates control data that is indicative of the size of theoutput, and a control unit that provides the control data to the secondoperation, wherein the second operation processes the input based on thecontrol data.

While the present invention is disclosed by reference to the preferredembodiments and examples detailed above, it is to be understood thatthese examples are intended in an illustrative rather than in a limitingsense. It is contemplated that modifications and combinations willreadily occur to those skilled in the art, which modifications andcombinations will be within the spirit of the invention and the scope ofthe following claims.

As will be appreciated by those of ordinary skill in the art, aspects ofthe presented technology may be embodied as a system, device, method, orcomputer program product apparatus. Accordingly, elements of the presentdisclosure may be implemented entirely in hardware, entirely in software(including firmware, resident software, micro-code, or the like) or insoftware and hardware that may all generally be referred to herein as a“apparatus,” “circuit,” “circuitry,” “module,” “computer,” “logic,”“FPGA,” “unit,” “system,” or other terms. Furthermore, aspects of thepresented technology may take the form of a computer program productembodied in one or more computer-readable medium(s) having computerprogram code stored thereon. The phrases “computer program code” and“instructions” both explicitly include configuration information for aCGRA, an FPGA, or other programmable logic as well as traditional binarycomputer instructions, and the term “processor” explicitly includeslogic in a CGRA, an FPGA, or other programmable logic configured by theconfiguration information in addition to a traditional processing core.Furthermore, “executed” instructions explicitly includes electroniccircuitry of a CGRA, an FPGA, or other programmable logic performing thefunctions for which they are configured by configuration informationloaded from a storage medium as well as serial or parallel execution ofinstructions by a traditional processing core.

Any combination of one or more computer-readable storage medium(s) maybe utilized. A computer-readable storage medium may be embodied as, forexample, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or other like storagedevices known to those of ordinary skill in the art, or any suitablecombination of computer-readable storage mediums described herein. Inthe context of this document, a computer-readable storage medium may beany tangible medium that can contain, or store, a program and/or datafor use by or in connection with an instruction execution system,apparatus, or device. Even if the data in the computer-readable storagemedium requires action to maintain the storage of data, such as in atraditional semiconductor-based dynamic random-access memory, the datastorage in a computer-readable storage medium can be considered to benon-transitory. A computer data transmission medium, such as atransmission line, a coaxial cable, a radio-frequency carrier, and thelike, may also be able to store data, although any data storage in adata transmission medium can be said to be transitory storage.Nonetheless, a computer-readable storage medium, as the term is usedherein, does not include a computer data transmission medium.

Computer program code for carrying out operations for aspects of thepresent technology may be written in any combination of one or moreprogramming languages, including object-oriented programming languagessuch as Java, Python, C++, or the like, conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages, or low-level computer languages, such as assemblylanguage or microcode. In addition, the computer program code may bewritten in VHDL, Verilog, or another hardware description language togenerate configuration instructions for an FPGA, CGRA IC, or otherprogrammable logic. The computer program code if converted into anexecutable form and loaded onto a computer, FPGA, CGRA IC, or otherprogrammable apparatus, produces a computer implemented method. Theinstructions which execute on the computer, FPGA, CGRA IC, or otherprogrammable apparatus may provide the mechanism for implementing someor all of the functions/acts specified in the flowchart and/or blockdiagram block or blocks. In accordance with various implementations, thecomputer program code may execute entirely on the user's device, partlyon the user's device and partly on a remote device, or entirely on theremote device, such as a cloud-based server. In the latter scenario, theremote device may be connected to the user's device through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider). Thecomputer program code stored in/on (i.e. embodied therewith) thenon-transitory computer-readable medium produces an article ofmanufacture.

The computer program code, if executed by a processor, causes physicalchanges in the electronic devices of the processor which change thephysical flow of electrons through the devices. This alters theconnections between devices which changes the functionality of thecircuit. For example, if two transistors in a processor are wired toperform a multiplexing operation under control of the computer programcode, if a first computer instruction is executed, electrons from afirst source flow through the first transistor to a destination, but ifa different computer instruction is executed, electrons from the firstsource are blocked from reaching the destination, but electrons from asecond source are allowed to flow through the second transistor to thedestination. So, a processor programmed to perform a task is transformedfrom what the processor was before being programmed to perform thattask, much like a physical plumbing system with different valves can becontrolled to change the physical flow of a fluid.

Example 1 is a data processing system for implementing operations thatgenerate a dynamically-sized output. The data processing systemcomprises a reconfigurable processor configured to implement: a firstoperation that generates an output, wherein a size of the output isunknown during a configuration phase; a second operation that receivesthe output of the first operation as an input; a recording unit thatgenerates control data that is indicative of the size of the output; anda control unit that provides the control data to the second operation,wherein the second operation processes the input based on the controldata.

In Example 2, the reconfigurable processor of Example 1 comprises arraysof coarse-grained reconfigurable (CGR) units that implement the firstand second operations.

In Example 3, the recording unit of Example 1 generates the control datawhile the first operation generates the output.

In Example 4, the recording unit of Example 3 comprises a counter thatcounts a number of elements in the output to generate the control data.

In Example 5, the counter of Example 4 increments conditionally on apredicate of the first operation.

In Example 6, the control unit of Example 1 fetches the control datafrom the recording unit.

In Example 7, the data processing system of Example 6 further comprisesexternal memory that is coupled to the reconfigurable processor, whereinthe control unit stores the control data in the external memory, andwherein the second operation retrieves the control data from theexternal memory.

In Example 8, the reconfigurable processor of Example 1 is furtherconfigured to implement a synchronization unit that informs the secondoperation when the first operation has generated the output.

In Example 9, the reconfigurable processor of Example 1 is furtherconfigured to store the output in a buffer during a write operation, andwherein the control unit directs the second operation during a readoperation following the write operation to read data as the input fromthe buffer that was stored during the write operation.

In Example 10, the reconfigurable processor of Example 9 is configuredto enable the first operation to write a first portion of the output toa first portion of the buffer, while the second operation reads a firstportion of the input that is different than the first portion of theoutput from a second portion of the buffer that is different than thefirst portion of the buffer.

In Example 11, the reconfigurable processor of Example 10 is configuredto enable the first operation to write, when the first operation hasfinished writing the first portion of the output and the secondoperation has finished reading the first portion of the input, a secondportion of the output to the second portion of the buffer, while thesecond operation reads the first portion of the output from the firstportion of the buffer as a second portion of the input.

In Example 12, the data processing system of Example 1 further comprisesa compiler that generates configuration data for configuring thereconfigurable processor to implement the first operation, the secondoperation, the recording unit, and the control unit.

In Example 13, the output comprises a number of elements that is smallerthan or equal to a predetermined maximum number of elements, and thecompiler of Example 12 generates in the configuration data a firstconnection for the output and a second connection for the input, whereineach one of the first and second connections suitable for a transmissionof the predetermined maximum number of elements.

In Example 14, the data processing system of Example 1 further comprisesruntime logic that is configured to program the reconfigurable processorwith configuration data such that the reconfigurable processorimplements the first operation, the second operation, the recordingunit, and the control unit.

Example 15 is a method of operating a data processing system thatcomprises a reconfigurable processor, comprising configuring thereconfigurable processor such that the reconfigurable processorimplements: a first operation that generates an output, wherein a sizeof the output is unknown during a configuration phase, a secondoperation that receives the output of the first operation as an input, arecording unit that generates control data that is indicative of thesize of the output, and a control unit that provides the control data tothe second operation, wherein the second operation processes the inputbased on the control data.

In Example 16, the method of Example 15 further comprises configuringthe reconfigurable processor such that the reconfigurable processorimplements a synchronization unit that informs the second operation whenthe first operation has generated the output.

In Example 17, the data processing system of Example 15 furthercomprises a compiler, and the method further comprises generating, withthe compiler, configuration data for configuring the reconfigurableprocessor to implement the first operation, the second operation, therecording unit, and the control unit.

In Example 18, the output comprises a number of elements that is smallerthan or equal to a predetermined maximum number of elements, and themethod of Example 17 further comprises generating, with the compiler, inthe configuration data a first connection for the output and a secondconnection for the input, wherein each one of the first and secondconnections is able to transport the predetermined maximum number ofelements.

In Example 19, the data processing system of Example 15 furthercomprises runtime logic, and wherein configuring the reconfigurableprocessor further comprises: programming, with the runtime logic, thereconfigurable processor with configuration data.

Example 20 is a non-transitory computer-readable storage mediumincluding instructions that, when executed by a processing unit, causethe processing unit to operate a data processing system that comprises areconfigurable processor, the instructions comprising: configuring thereconfigurable processor with configuration data such that thereconfigurable processor implements: a first operation that generates anoutput, wherein a size of the output is unknown, a second operation thatreceives the output of the first operation as an input, a recording unitthat generates control data that is indicative of the size of theoutput, and a control unit that provides the control data to the secondoperation, wherein the second operation processes the input based on thecontrol data.

What is claimed is:
 1. A data processing system for implementingoperations that generate a dynamically-sized output, comprising: areconfigurable processor configured to implement: a first operation thatgenerates an output, wherein a size of the output is unknown during aconfiguration phase; a second operation that receives the output of thefirst operation as an input; a recording unit that generates controldata that is indicative of the size of the output; and a control unitthat provides the control data to the second operation, wherein thesecond operation processes the input based on the control data.
 2. Thedata processing system of claim 1, wherein the reconfigurable processorcomprises arrays of coarse-grained reconfigurable (CGR) units thatimplement the first and second operations.
 3. The data processing systemof claim 1, wherein the recording unit generates the control data whilethe first operation generates the output.
 4. The data processing systemof claim 3, wherein the recording unit comprises: a counter that countsa number of elements in the output to generate the control data.
 5. Thedata processing system of claim 4, wherein the counter incrementsconditionally based on a predicate of the first operation.
 6. The dataprocessing system of claim 1, wherein the control unit fetches thecontrol data from the recording unit.
 7. The data processing system ofclaim 6, further comprising: external memory that is coupled to thereconfigurable processor, wherein the control unit stores the controldata in the external memory, and wherein the second operation retrievesthe control data from the external memory.
 8. The data processing systemof claim 1, wherein the reconfigurable processor is further configuredto implement a synchronization unit that informs the second operationwhen the first operation has generated the output.
 9. The dataprocessing system of claim 1, wherein the reconfigurable processor isfurther configured to store the output in a buffer during a writeoperation, and wherein the control unit directs the second operationduring a read operation following the write operation to read data asthe input from the buffer that was stored during the write operation.10. The data processing system of claim 9, wherein the reconfigurableprocessor is configured to enable the first operation to write a firstportion of the output to a first portion of the buffer, while the secondoperation reads a first portion of the input that is different than thefirst portion of the output from a second portion of the buffer that isdifferent than the first portion of the buffer.
 11. The data processingsystem of claim 10, wherein the reconfigurable processor is configuredto enable the first operation to write, when the first operation hasfinished writing the first portion of the output and the secondoperation has finished reading the first portion of the input, a secondportion of the output to the second portion of the buffer, while thesecond operation reads the first portion of the output from the firstportion of the buffer as a second portion of the input.
 12. The dataprocessing system of claim 1, further comprising: a compiler thatgenerates configuration data for configuring the reconfigurableprocessor to implement the first operation, the second operation, therecording unit, and the control unit.
 13. The data processing system ofclaim 12, wherein the output comprises a number of elements that issmaller than or equal to a predetermined maximum number of elements, andwherein the compiler generates in the configuration data a firstconnection for the output and a second connection for the input, whereineach one of the first and second connections is suitable for atransmission of the predetermined maximum number of elements.
 14. Thedata processing system of claim 1, further comprising: runtime logicthat is configured to program the reconfigurable processor withconfiguration data such that the reconfigurable processor implements thefirst operation, the second operation, the recording unit, and thecontrol unit.
 15. A method of operating a data processing system thatcomprises a reconfigurable processor, comprising: configuring thereconfigurable processor such that the reconfigurable processorimplements: a first operation that generates an output, wherein a sizeof the output is unknown during a configuration phase, a secondoperation that receives the output of the first operation as an input, arecording unit that generates control data that is indicative of thesize of the output, and a control unit that provides the control data tothe second operation, wherein the second operation processes the inputbased on the control data.
 16. The method of claim 15, furthercomprising: configuring the reconfigurable processor such that thereconfigurable processor implements a synchronization unit that informsthe second operation when the first operation has generated the output.17. The method of claim 15, wherein the data processing system furthercomprises a compiler, further comprising: generating, with the compiler,configuration data for configuring the reconfigurable processor toimplement the first operation, the second operation, the recording unit,and the control unit.
 18. The method of claim 17, wherein the outputcomprises a number of elements that is smaller than or equal to apredetermined maximum number of elements, the method further comprising:generating, with the compiler, in the configuration data a firstconnection for the output and a second connection for the input, whereineach one of the first and second connections is able to transport thepredetermined maximum number of elements.
 19. The method of claim 15,wherein the data processing system further comprises runtime logic, andwherein configuring the reconfigurable processor further comprises:programming, with the runtime logic, the reconfigurable processor withconfiguration data.
 20. A non-transitory computer-readable storagemedium including instructions that, when executed by a processing unit,cause the processing unit to operate a data processing system thatcomprises a reconfigurable processor, the instructions comprising:configuring the reconfigurable processor with configuration data suchthat the reconfigurable processor implements: a first operation thatgenerates an output, wherein a size of the output is unknown, a secondoperation that receives the output of the first operation as an input, arecording unit that generates control data that is indicative of thesize of the output, and a control unit that provides the control data tothe second operation, wherein the second operation processes the inputbased on the control data.